Late multimodal fusion for image and audio music transcription

Music transcription, which deals with the conversion of music sources into a structured digital format, is key problem for Information Retrieval (MIR). When addressing this challenge in computational terms, MIR community follows two lines research: documents, case Optical Recognition (OMR), or audio recordings, Automatic Transcription (AMT). The different nature aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition terms sequence labeling tasks leads common output representation, enables research on combined paradigm. In respect, multimodal image and transcription comprises effectively combining information conveyed by modalities. work, we explore question at late-fusion level: study four combination approaches order merge, first time, hypotheses regarding end-to-end OMR AMT systems lattice-based search space. results obtained series performance scenarios–in corresponding single-modality models yield error rates–showed interesting benefits approaches. addition, strategies considered significantly improve unimodal standard recognition